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ABSTRACT 



How can students develop shared criteria for problems that 
have no "right" answer? Ill-structured problems of this sort are called 
design problems. Like portfolio projects, these problems are difficult to 
evaluate for both teachers and students. This investigation contrasts two 
methods for developing shared criteria for project evaluation. Elements of 
self-criticism and peer review appear in both methods with levels of 
collaboration altered between the two conditions. The investigation used the 
Scaffolded Knowledge Integration (SKI) framework to guide the curriculum 
design. The two approaches were an autonomous condition in which students 
worked in pairs on a single design and then critiqued their own designs and a 
collaborative condition in which students create separate initial designs and 
then merged them into one. Eighth graders (n=148) used the Knowledge 
Integration Environment (KIE) to design passive solar energy efficient houses 
for the desert. Results show that in both conditions students develop shared 
criteria in the sense that, as a group, they evaluate pieces of evidence to 
support their designs. However, the collaborative condition makes the need to 
evaluate alternatives visible to students. This increase in visibility may 
partially account for the increase in principled selection among alternatives 
in the collaborative condition. This effect appears to be strongest for 
students who do not distinguish between ideas such as heat and temperature. 
Ultimately, building shared criteria, through collaboration for some students 
and through autonomous activities for others, can scaffold students as they 
become more autonomous critical thinkers. (Contains two figures.) (SLD) 
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How can students develop shared criteria especially for problems that have no 
"right" answer? Such ill-structured problems are typically called design problems. They 
can be identified by their open-endedness, the problem structuring they require, and the 
iterative refinement that characterizes solution paths., Like portfolio projects, these 
problems are difficult to evaluate for both teachers and students. Often success is measured 
by the relative fit between the solution and the criteria selected to define the problem. 
Therefore if students select different criteria, a uniform metric for evaluation cannot be 
used. As educators and researchers, we need to encourage educational approaches that 
help students develop shared criteria not just for design projects but for analysis of 
scientific methods. 

Objective 

This investigation contrasts two methods for developing shared criteria for project 
evaluation. Elements of self-critique and peer review appear in both methods with the 
levels of collaboration altered between the two conditions. 

Theoretical Perspective 

The investigation uses the Scaffolded Knowledge Integration (SKI) framework 
(Linn, 1995) to guide the curriculum design. One aspect of this framework involves 
making thinking visible. Vygotsky's social theory of learning which suggests that we learn 
through the internalization of observed behavior guided the design of the collaborative 
condition. We anticipate that the development of shared criteria can occur through a 
process of peer review that is reinforced or reinterpreted through self-critique. In addition, 
the opportunity for self-analysis provides a chance for students to become more 
autonomous learners. 

To help student develop shared criteria, we use several approaches to make thinking 
visible. For example, students complete worksheets designed to encourage iterative 
refinement of the problem. These worksheets involve specifying temperature regulation 
approaches, selecting relevant structural aspects of the dwelling (e.g., windows, roofs, 
walls, materials, color, and orientation), creating functional representations of the heating- 
cooling cycle, and drawing graphical representations of heat flow at different times of day. 
Other approaches we use include an online collaborative search page for pooling comments 
about Internet resources and an online peer review form for final designs. 

This paper focuses on the interaction between two of these solutions that have 
resulted in radical revisions to designs and by extension the criteria used to evaluate them. 
These two approaches are: (a) an autonomous condition where students work in pairs on a 
single design and then critique their own designs and (b) a collaborative condition where 
students create separate initial designs and then merge them into one. Our hypothesis is 
that students that are unable to generate design alternatives will have trouble with self- 
critique. Collaboration can help these students expand their problem definition and build 
shared criteria by making thinking visible through self-explanation. 



Methods 



In this experiment using the Knowledge Integration Environment (KIE), eighth 
graders (N=148) design passive-solar, energy efficient houses for the desert. The house 
design project is a capstone activity following a semester on thermodynamics. Prior to the 
house design project, students gain experience using scientific principles to make 
predictions about experimental situations. They critique evidence based on its reliability, 
credibility, and source helping prepare them to search the Internet for evidence to support 
their house designs. 

Results & Discussion 

The majority of the teams (~90%) completed the two-week project with many of the 
students coming in during lunch and after school to work on their designs. Students in the 
collaborative condition tended to consider alternatives more carefully. Beyond the 
experimental effects, factors such as the student's disposition and the nature of the 
interaction between partners contributed to the success of the projects. 

Overall Analysis. Overall, there was a significant correlation (F(148)=4.787, 
pc.OOl) between the number of alternatives considered and the innovation of the design. 
These numbers provide an indication that innovation may be linked to the rate of revision. 

Alternatives Considered. Surprisingly, students who had to merge designs in the 
collaborative condition did not consider more design alternatives than those in the 
autonomous condition. However, the number of design alternatives considered and 
discarded based on scientific principles (or evidence found on the Internet) was higher for 
the collaborative group than for the autonomous group. 

Experimental Effects. Initial coding of pre and post-tests suggest that the 
collaborative condition benefited students with confused terminology for the heating and 
cooling cycle in the desert (e.g., students that did not differentiate between heat and 
temperature or insulation and conduction.) The autonomous condition benefited students 
with descriptive but unmechanistic explanations (e.g., "it takes longer for the ground to 
heat up.") Unfortunately, it is difficult to assign causality given die complex nature of 
social dynamics and the numerous factors involved in classroom activities. 

Information Constraining Designs. An additional factor, the range of available 
information, constrains the process of selecting criteria In the majority of cases where 
students introduced new criteria for a design, information from our online house design 
library or the Internet drove the selection process. Figure 1 shows the two initial designs 
and the final design for a group that refined ideas about ventilation by adding a retractable 
solar panel. The house rotates in the evening so that the passive solar panels, colored black 
for optimum energy conversion, can receive light from the setting sun. 





Figure 1. Refinement of a ventilation strategy. 

The students looked first for information on windows. Then realizing that they were the 
weakest link in most houses, they considered additional alternatives such as using trees to 
shade the house or solar panels as walls. The solution they reached was to raise the solar 
panels above the house and circulate the water through the basement for radiant heating. 

Knowledge Building. More qualitative measures of the effect of collaboration on 
the design process indicate that students respond differently to peer review depending on 
their initial level of understanding and their methods of self-explanation. For example, one 
pair of students in the collaborative condition started with similar ideas about why the 
outside temperature in the desert peaks before the ground temperature. Specifically, they 
believed that the sand or dirt took a while to absorb the heat energy. In the pre-test, neither 
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student distinguished between heat and temperature. However, one student attempted to 
explain situations by referring to previous labs and prototypes: 

I will use the burner example again. If you turn a burner on high - then it heats 
up very quickly. But because it is heating up a material it takes longer for the 
heat to filtrate through a substance and therefore raise the temperature. So if 
the burner was on high - then you would have to wait awhile before you see the 
water start to bubble or heat up. 

The students located two Internet sites that lead to their consideration of light converting to 
heat energy as a possible factor: a site showing the reflectance index for roofing materials 
and a site showing how location affects climate. The evolution of their designs from initial 
sketches through the final design appears in figure 2. 





Figure 2. Merging of two students drawings into a final design. 

Structurally the students have combined their designs incorporating new features based on 
the available information. There is a rock to the east of the house to reduce direct sunlight 
in the morning. The color of the house is white "because in a lab that we did, we learned 
that white scatters light more than a black house would. This is especially good for houses 
in the desert because there is a lot of light that would change to heat energy." 

In the final design report, the pair of students refers to previous experiences and 
scientific principles to explain their reasoning. Closer examination of the post-test reveals 
that the students, while having similar experiences, exhibit different degrees of knowledge 
building possibly due to the way in which they are processing information (see Chen, 
Burtis, & Bereiter, 1997.) The student that is trying to explain things in terms of 
prototypes has shifted her explanation for why the outside temperature peaks before the 
ground temperature: 

The ground temperature heats up slower than the air temperature because the 
heat energy has to flow through the air before it reaches the ground. It also 
takes time for the ground to absorb the heat energy. Like if you shined a light 
bulb onto a test tube of sand — the sand wouldn't heat up to its peak 
temperature in a half of a second, it would take time. 

While this student has moved from a conduction to a light conversion model, the other 
student does not consider light as a factor in the equation. Granted this problem is complex 
and dependent on a wide range of factors. However, the interesting dimension of the 
problem is that collaborative work has benefits and trade-offs that appear to be linked to the 
students' previous level of understanding and to their methods for explaining phenomena 

Educational Implications 

We are just beginning to understand the benefits and trade-offs of collaboration and 
the role it plays in building shared criteria. In both conditions, students develop shared 
criteria in the sense that, as a group, they evaluate pieces of evidence to support their 
designs. However, the collaborative condition makes the need to evaluate alternatives 
visible to students. This increased visibility may partially account for the increase in 
principled selection among alternatives in the collaborative condition. This effect appears to 
be strongest for students that do not distinguish between ideas such as heat and 
temperature. Ultimately, building shared criteria, through collaboration for some students 
and through autonomous activities for others, can scaffold students as they become more 
autonomous critical thinkers. 
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